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PREFACE 


The  Test  Plan  is  the  fifth  in  a  series  of  system  development  documents 
published  on  the  Local  Automation  Model  project.  To  date,  we  have  defined  the 
system  requirements,  developed  a  conceptual  design,  established  functional  and 
performance  specifications,  surveyed  existing  library  automation  software 
products,  and  recommended  six  of  these  packages  for  comparison. 

Available  for  implementation  by  Department  of  Defense  (DoD)  technical 
libraries,  the  system  will  integrate  local  collection  management  functions  — 
retrieval,  cataloging,  and  circulation  management  and  control  —  with  system 
features  required  to  support  resource  sharing.  Resource  sharing  includes 
shared  cataloging  by  individual  technical  libraries  into  the  Technical  Reports 
data  base,  a  central  repository  for  bibliographic  citations  to  over 
1.5  million  technical  reports  prepared  by  and  for  the  DoD.  Resource  sharing 
also  encompasses  access  to  external  bibliographic  resources  —  both  Government 
and  commercial  —  using  a  single  system  and  command  language  for  information 
retrieval.  A  prototype  of  the  system  is  planned  for  implementation  at  the 
Defense  Nuclear  Agency  during  fiscal  year  1985  to  demonstrate  the  concept  as 
well  as  provide  an  evaluation  period  prior  to  acquiring  production  systems. 

Both  the  prototype  and  production  systems  will  be  implemented  using 
commercially  available  software  for  local  collection  management.  The  produc¬ 
tion  system  software  will  be  acquired  through  competitive  procurement.  This 
Test  Plan  describes  the  benchmark  procedures,  assigns  organizational 
responsibility,  and  lists  the  evaluation  and  scoring  criteria  used  for 
selecting  the  prototype  system  from  among  the  six  candidate  software  packages. 


SECTION  1 .  GENERAL 


1.1  Purpose. 

This  Test  Plan  for  the  Local  Automation  Model  (LAM)  is  written  to: 

-  Provide  a  description  of  the  overall  concept  for  conducting  software 
benchmarking; 

-  Describe  the  procedures  for  Phase  I  benchmarking  of  selected  software 
packages ; 

-  Establish  a  comprehensive  test  plan  and  communicate  to  the  test  par¬ 
ticipants  the  nature  and  extent  of  the  tests  deemed  necessary  to 
provide  a  basis  for  evaluating  and  selecting  software  for  prototype 
system  implementation; 

-  Coordinate  with  test  participants  a  schedule  of  events,  a  specifica¬ 
tion  of  equipment  and  organizational  requirements,  and  the  methods  for 
testing;  and 

-  Provide  a  written  record  of  the  evaluation  criteria  used  for  assessing 
software  package  performance  and  selecting  a  package  for  prototype 
system  implementation. 

This  Test  Plan  describes  the  overall  approach  for  benchmarking 
(Section  2)  and  addresses  the  procedures,  evaluation  criteria,  and  schedule 
for  Phase  I  benchmarking  (Sections  3  and  4).  As  used  throughout  this  docu¬ 
ment,  the  term  benchmarking  refers  to  a  software  testing  and  evaluation 
process  consisting  of  (1)  an  assessment  of  the  functional  completeness  of  a 
software  package  (i.e.,  does  the  package  perform  the  functions  required  for 
system  implementation)  and  (2)  an  assessment  of  the  relative  performance  of 
six  candidate  software  packages. 

Phase  I  of  the  benchmark  covers  the  installation  and  evaluation  of 
selected  software  packages  independent  of  other  software  required  for 
prototype  system  implementation.  Phase  II  encompasses  integration  of  the 
software  packages  with  the  intelligent  gateway  software.  This  intelligent 
gateway  software,  developed  by  Lawrence  Livermore  National  Laboratory  (LLNL) , 
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is  important  to  meet  the  LAM  requirement  to  communicate  with  the  Defense 


Technical  Information  Center  (DTIC)  data  bases.  Phase  III  covers  an 
assessment  of  the  performance  of  the  software  packages  integrated  with  the 
intelligent  gateway  software.  Phase  III  test  procedures  and  evaluation 
criteria  will  be  published  separately. 

1.2  Project  References. 

The  LAM  encompasses  a  system  design,  development,  and  evaluation  project 
sponsored  by  the  DTIC  located  at  Cameron  Station  in  Alexandria,  Virginia.  The 
requirements  for  and  design  of  the  system  are  documented  in: 

-  Local  Automation  Model:  Conceptual  Design  Document,  Logistics  Manage¬ 
ment  Institute,  April  1983; 

-  Local  Automation  Model:  Functional  Description,  Logistics  Management 
Institute,  October  1983; 

-  Local  Automation  Model:  System  Specification,  Logistics  Management 
Institute,  February  1984;  and 

-  Local  Automation  Model:  Assessment  of  Library  Software  Availability, 
Logistics  Management  Institute,  September  1984. 

The  prototype  system  —  planned  for  implementation  at  the  Defense  Nuclear 
Agency  (DNA)  in  Alexandria,  Virginia  --  will  provide  the  opportunity  to  demon¬ 
strate  and  evaluate  an  automated  library  system  with  special  features  for 
bibliographic  information  sharing.  The  system  will  support  conventional 
collection-handling  capabilities  such  as  original  cataloging  and  citation 
retrieval.  In  addition,  the  system  will  facilitate  information  sharing 
between  Department  of  Defense  (DoD)  technical  libraries  and  DTIC  by  incor¬ 
porating  intelligent  gateway  processing  capabilities. 

Gateway  capabilities  required  for  the  system  include:  (1)  automatic 
searching  of  both  the  local  technical  library  catalog  and  the  DTIC  Technical 
Reports  (TR)  data  base  using  a  single  search  language  and  format,  (2)  down¬ 
loading  of  information  from  the  TR  data  base  to  the  local  system,  and 


(3)  machine-aided  translation  of  locally  created  catalog  citations  into  a 
format  acceptable  for  entry  in  the  TR  data  base.  Thus,  with  one  system  and 
one  set  of  commands,  a  technical  library  can:  (1)  maintain  and  expand  a 
catalog  tailored  to  local  needs,  (2)  access  the  wealth  of  information  con¬ 
tained  in  the  TR  data  base,  and  (3)  contribute  directly  to  the  timely  dis¬ 
semination  of  scientific  and  technical  information  via  direct  cataloging  in 
the  TR  data  base. 

The  system  is  intended  for  implementation  by  technical  libraries  partici¬ 
pating  in  the  Shared  Bibliographic  Input  Network  (SBIN)  sponsored  by  DTIC. 
(SBIN  was  established  initially  as  an  experiment  in  shared  cataloging  and  has 
since  been  established  as  an  on-going  OTIC  program.)  At  present,  in  addition 
to  providing  inputs  to  a  local  catalog,  SBIN  member  libraries  catalog  in  the 
TR  data  base  using  a  separate  input  system  supported  by  the  Defense  RDT&E 
On-Line  System  (DROLS) .  This  procedure  results  in  a  duplication  of  manual  and 
intellectual  effort  and  serves  as  a  disincentive  for  participating  in  shared 
cataloging.  In  large  part,  initiation  of  and  requirements  for  the  LAM  project 
reflect  the  need  to  reduce  the  burden  of  effort  placed  on  SBIN  members  and 
thereby  promote  information  sharing. 

Included  in  the  LAM  project  is  the  development  of  an  acquisition  strategy 
and  plan  for  a  production  system.  Experience  gained  through  operation  and 
evaluation  of  the  prototype  system  will  contribute  directly  to  development  of 
the  acquisition  plan.  Since  publication  of  the  Functional  Description  and 
System  Specification,  libraries  and  information  centers  outside  of  SBIN  have 
expressed  interest  in  implementing  the  system.  On  the  basis  of  this  interest, 
DTIC  will  make  the  production  system  available  to  other  DoD  libraries. 

The  prototype  and  production  systems  will  be  implemented  using  com¬ 
mercially  available  library  software.  The  gateway  features  available  in  the 


prototype  system  will  be  provided  through  adaptation  of  a  subset  of  the  Inte¬ 
grated  Information  System  (IIS)  developed  by  the  Technology  Information  System 
(TIS)  group  located  at  LLNL.  Commercial  software  for  the  prototype  system 
will  be  selected  through  benchmarking  as  documented  in  this  Test  Plan. 
Competitive  bids  will  be  solicited  for  acquisition  of  the  production  system. 
1.3  Terms  and  Abbreviations. 


The  following  terms,  acronyms,  and  abbreviations  are  used  in  this 
document : 

-  ASCII:  American  Standard  Code  for  Information  Interchange; 

-  ATLAS:  Automated  Technical  Library  Accession  System  —  the  automated 
catalog  of  technical  report  bibliographic  citations  currently  main¬ 
tained  by  the  Defense  Nuclear  Agency; 

-  DNA:  Defense  Nuclear  Agency; 

-  DoD:  Department  of  Defense; 

-  DROPS :  Defense  RDT&E  On-Line  System; 

-  DTIC:  Defense  Technical  Information  Center; 

-  IIS:  Integrated  Information  System  —  an  intelligent  gateway 

developed  and  supported  exclusively  by  the  Technology  Information 
System  group  at  Lawrence  Livermore  National  Laboratory; 

-  LAM:  Local  Automation  Model  —  a  project  sponsored  by  the  Defense 
Technical  Information  Center  for  demonstrating,  evaluating,  and 
acquiring  an  integrated  library  system  encompassing  local  collection, 
management,  and  access  to  external  bibliographic  resources; 

-  LLNL:  Lawrence  Livermore  National  Laboratory  —  a  Department  of 

Energy- funded,  contractor-operated  research  and  development  laboratory 
located  in  Livermore,  California; 

-  RDT&E:  Research,  Development,  Test,  and  Evaluation; 

-  SBIN:  Shared  Bibliographic  Input  Network; 

-  TIS:  Technology  Information  System  —  used  to  describe  both  the  work 
on  advanced  information-handling  technology  and  the  organizational 
element  (group)  performing  this  work  at  LLNL;  and 

-  TR  data  base:  The  Technical  Reports  data  base  operated  and  maintained 
by  DTIC  containing  more  than  1  million  citations  to  reports  published 
or  sponsored  by  DoD. 
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SECTION  2.  BENCHMARKING  PLAN 


2.1  Introduction. 

The  applications  software  for  the  LAM  consists  of  two  major  components: 
(1)  a  commercial  library  software  package  supporting  local  collection  cata¬ 
loging,  retrieval,  and  circulation  management  and  control  and  (2)  an  intelli¬ 
gent  gateway  providing  access  to  the  DTIC  TR  data  base  for  cataloging  and 
retrieval.  The  gateway  also  provides  postprocessing  capabilities  such  as 
transferring  TR  data  base  citations  to  the  local  system,  translating  and 
merging  citations  from  both  sources  into  a  common  format,  eliminating  dupli¬ 
cate  or  unwanted  citations,  citation  sorting,  and  reordering  citation  contents 
(data  fields). 

The  commercial  library  software  package  used  for  prototype  system  imple¬ 
mentation  will  be  selected  through  benchmarking.  Benchmarking  will  be 
conducted  in  three  phases.  Phase  I  benchmarking  —  the  subject  of  this  Test 
Plan  —  consists  of  independent  testing  and  evaluation  of  six  candidate 
software  packages.  Its  objective  is  to  confirm  the  capabilities  of  each 
package  for  providing  the  functions  required  for  local  collection  management. 

Phase  II  of  the  benchmark  involves  assessing  the  technical  implications 
and  feasibility  of  integrating  the  software  packages  with  intelligent  gateway 
software.  Each  package  satisfactorily  completing  Phase  I  benchmarking  will  be 
subjected  to  this  assessment.  Where  feasible,  the  packages  will  be  integrated 
with  the  gateway  software.  Requirements  for  integrating  gateway  software  with 
the  commercial  packages  will  be  specified  in  Local  Automation  Model :  Program 
Specification  for  Cataloging  and  Retrieval  User  Access  published  by  the 
Logistics  Management  Institute  (LMI).  That  document  will  describe  the 
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requirements  and  specifications  for  facilitating  user  access  to  the  integrated 
system. 

Phase  III  of  the  benchmarking  consists  of  evaluating  system  performance 
of  the  integrated  software  components;  i.e.,  the  commercial  packages  linked 
with  the  intelligent  gateway  through  a  user  interface.  The  objective  of 
Phase  III  is  selection  of  an  integrated  package  for  prototype  system  implemen¬ 
tation.  A  secondary  goal  of  Phase  III  is  testing  and  refinement  of  the  user 
interface  for  accessing  the  system. 

2.2  Pretest  Activity. 

The  following  activities  must  be  completed  in  preparation  for  benchmark¬ 
ing:  (1)  selection,  acquisition,  and  installation  of  the  candidate  packages 

and  (2)  test  catalog  (data  base)  development.  Each  pretest  activity  is 
discussed  in  this  subsection. 

Assessment  of  Library  Software  Availability.  A  survey  was  conducted  to 
assess  the  availability  of  commercial  library  software  packages  for 
performance  benchmarking  and  to  select  packages  for  testing.  The  following 
six  packages,  marketed  by  the  companies  indicated,  were  selected: 

BIBT.IOTECH  --  Comstow  Information  Service; 

BRS/SEARCH  —  Bibliographic  Retrieval  Service; 

DATALIB  —  M/A-COM  Sigma  Data,  Inc.; 

-  LS/2000  —  OCLC,  Inc.; 

MINISIS  --  Systemhouse,  Inc.;  and 

ULISYS  --  Universal  Library  Systems. 

A  subset  of  the  IIS  will  be  used  to  provide  the  gateway  processing  capa¬ 
bilities  required  for  the  system.  The  IIS  is  unique  in  that  it  offers  the 
broad  range  of  postprocessing  and  computer-to-computer  data  transfer  features 
essential  for  LAM  operations. 
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Software  Acquisition  and  Installation.  The  TIS  group  at  LLNL  will 
arrange  for  temporary  use  of  each  of  the  software  packages  selected  for  bench¬ 
marking.  First  preference  will  be  to  install  each  package  locally  at  LLNL. 
Alternatively,  access  to  the  packages  via  dial-up  telephone  lines  and  termi¬ 
nals  is  acceptable  for  testing.  All  test  participants  (see  Section  2.4.4) 
must  have  access  to  the  packages. 

Test  Catalog  Development.  A  catalog  of  technical  report  citations  will 
be  prepared  for  use  during  benchmarking.  The  catalog  will  be  composed  of 
unclassified,  publicly  releasable  citations  derived  from  the  DTIC  TR  data  base 
and  the  DNA  Automated  Technical  Library  Accession  System  (ATLAS)  file. 
Five  hundred  citations  will  be  extracted  from  each  source,  written  on  magnetic 
tape,  and  transferred  to  LLNL.  LLNL  will  load  the  catalog  for  packages 
installed  at  the  Laboratory  and  will  provide  copies,  with  accompanying  dis¬ 
closure  limitation  restrictions,  to  those  vendors  offering  telephone  access  to 
software  packages. 

2.3  Testing  Schedule. 

Testing  will  take  place  at  four  locations:  LLNL,  DNA,  DTIC,  and  LMI. 
Test  participants  at  those  locations  will  use  dial-up  telephone  access  to  a 
computer  either  located  at  LLNL  (for  packages  installed  there)  or  provided  by 
a  vendor  (for  packages  not  installed  at  LLNL).  Benchmarking  will  take  place 
from  November  1984  through  April  1985.  Included  in  the  initial  stages  of  this 
schedule  is  the  development  of  a  representative  test  catalog  and  the 
installation  of  vendor-supplied  software  packages. 

2.4  Phase  I  --  Commercial  Package  Testing. 

Phase  I  benchmarking  will  consist  of  a  comparison  of  the  six  software 
packages  selected  during  the  software  availability  assessment.  Each  package 


will  be  exercised  to  determine  whether  it  offers  the  functions  and  features 


2-3 


specified  for  the  LAM  system.  In  addition,  test  participants  will  be  asked  to 
rank  the  six^packages  on  a  function-by-function  basis  so  that  one  package  can 
be  selected  for  prototype  system  implementation.  Members  of  the  DNA  technical 
library  staff  will  participate  in  this  phase  of  the  benchmarking  and  will  be 
responsible  for  exercising  each  software  package  as  outlined  in  Section  4  of 
this  Test  Plan.  Staff  members  from  the  LLNL  TIS  group  will  operate  the 
systems  subjected  to  benchmarking  on  LLNL  computers  and,  in  conjunction  with 
OTIC  staff  members  participating  in  the  test,  will  assess  the  technical  per¬ 
formance  and  suitability.  Staff  members  from  LMI  will  coordinate  the  testing, 
compile  the  test  results,  and  prepare  a  Test  Analysis  Report  documenting  those 
results. 

2.4.1  Milestone  Chart. 

Figure  2-1  depicts  the  schedule  for  Phase  I  benchmarking. 

2.4.2  Equipment  Requirements. 

Test  participants  are  responsible  for  providing  the  equipment  needed  for 
dial-up  access  to  vendor-provided  software  packages.  Section  2. 4. 6. 2,  "Site- 
Supplied  Materials,"  presents  a  description  of  equipment  requirements.  Test 
participants  must  also  request  access  rights  to  the  computers  used  for  bench¬ 
marking.  For  vendor-provided  computers,  access  will  be  coordinated  through 
LLNL;  for  access  to  the  LLNL  computer  and  vendor-provided  computers  used  for 
benchmarking,  requests  should  be  sent  to  Dr.  Hilary  Burton,  the  project  leader 
at  LLNL. 

Vendors  providing  dial-up  access  to  software  packages  for  benchmarking 
will  generally  permit  access  for  a  specified  period  of  time.  Schedules  estab¬ 
lished  by  these  vendors  will  be  coordinated  by  LLNL  and  will  be  distributed  to 
test  participants  as  supplements  to  the  Test  Plan  prior  to  commencement  of 


benchmarking. 
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For  packages  installed  at  LLNL,  the  Laboratory  generally  allows  dial-up 
access  to  their  computer  system  any  time  of  the  day.  If  LLNL  sets  aside 
specific  blocks  of  time  for  LAM  benchmarking  (to  reduce  contention  for 
resources  and  improve  system  responsiveness),  test  participants  will  be  noti¬ 
fied  by  supplements  to  the  Test  Plan. 

2.4.3  Software. 

Vendors  are  encouraged  to  provide  software  for  implementation  at  LLNL. 
Packages  provided  by  vendors  must  be  compatible  with  the  operating  environment 
supported  by  existing  Laboratory  computers  available  for  benchmarking.  If  a 
compatible  machine  is  not  available  at  LLNL,  vendors  may  provide  dial-up 
access  to  a  computer  running  their  package.  Access  to  packages  by  test  par¬ 
ticipants  must  be  available  throughout  the  period  of  time  scheduled  for 
Phase  I  benchmarking.  LLNL  is  responsible  for  acquiring  access  rights  to 
vendor  software.  Should  problems  arise  in  obtaining  vendor  participation  or 
access  to  software  for  benchmarking  purposes,  the  project  monitor  at  DTIC  will 
be  notified  and  alternative  courses  of  action  recommended. 

2.4.4  Personnel. 

The  DNA  will  be  responsible  for  providing  at  least  two  technical  library 
staff  members  to  be  test  participants.  The  DNA  is  encouraged  to  assign  addi¬ 
tional  staff  members  to  obtain  broader  participation  by  potential  system 
users . 

Staff  members  from  DTIC  will  provide  assessments  of  software  performance, 
functionality,  and  technical  suitability  for  implementing  the  prototype 
system. 

Test  participants  may  perform  their  evaluations  independently  of  other 
participants,  consistent  with  existing  assignments  and  job  responsibilities. 
However,  a  test  participant  must  complete  a  full  evaluation  of  all  packages  so 


that  a  valid  relative  performance  ranking  can  be  obtained.  That  is,  if  a  test 
participant  evaluates  cataloging  functions  and  features  (as  described  in 
Section  4),  then  those  functions  and  features  must  be  evaluated  in  every 
package  subjected  to  benchmarking. 

The  results  of  each  participant's  evaluation  will  be  a  relative  ranking 
of  all  systems  benchmarked  reflecting  the  relative  performance,  completeness, 
and  ease  of  use  demonstrated  by  each  package. 

Staff  members  from  LLNL  will  operate  the  systems  installed  at  the 
Laboratory  and  will  coordinate  with  all  vendors  concerning  access  to  software, 
equations  on  the  operation  of  packages,  and  test  catalog  development.  LLNL 
staff  members  will  provide  technical  assessments  of  package  performance,  ease 
of  operation,  and  overall  suitability  for  prototype  system  implementation. 
Particular  consideration  will  be  given  to  assessing  the  feasibility  of  inte¬ 
grating  IIS  software  with  the  commercial  packages  selected  for  benchmarking. 

Staff  members  from  LMI  will  be  responsible  for  compiling  results  and 
preparing  a  Test  Analysis  Report  summarizing  the  results.  LMI  staff  members 
will  also  evaluate  selected  features.  Benchmark  results  will  be  coordinated 
with  DNA  technical  library  staff  members,  DTIC  test  participants,  and  project 
members  from  LLNL. 

Recommendations  covering  organizational  responsibilities  for  testing  each 
of  the  system  functions  included  in  the  benchmark  are  discussed  in 
Section  4.1,  "Test  Description." 

2.4.5  Orientation  Plan. 

The  LAM  project  leader  for  LLNL  will  provide  test  participants  with  short 
instruction  sheets  explaining  how  to  use  each  package  subjected  to  benchmark¬ 
ing.  Specific  questions  on  the  operation  of  a  package  should  be  directed  to 
the  LLNL  project  leader  who  will  serve  as  the  central  point  of  contact  with 
the  participating  software  vendors. 


2.4.6  Test  Materials. 

Each  test  participant  will  receive  a  copy  of  this  Test  Plan.  The 
description  and  evaluation  factors  specified  in  Section  4  and  Appendices  A 
through  H  are  required  for  conducting  the  comparative  tests  selected  for  the 
LAM  benchmark.  This  Test  Plan  will  be  supplemented  as  required  to  assist  test 
participants  in  conducting  the  benchmarking.  The  passwords  and  user 
identification  codes  required  to  access  the  computers  on  which  the  six  soft¬ 
ware  packages  will  be  operated  will  be  provided  to  all  test  participants  in  a 
separate  memorandum.  The  passwords  and  access  codes  must  be  safeguarded  to 
prevent  unauthorized  use  of  the  software.  Schedules  of  software  and  computer 
availability  will  also  be  provided  as  software  installation  progresses. 

2.4.6. 1  Deliverable  Materials. 


For  each  package  made  available  for  benchmarking,  supporting  documenta¬ 
tion  such  as  user's  manuals  and  operations  guides  will  be  requested  from  the 
vendor.  If  made  available  for  installation  at  LLNL,  the  vendor  must  supply 
program  tapes  and  sufficient  documentation  to  permit  program  loading  and 
execution.  Vendor-provided  operations  guides  must  provide  sufficient  detail 
to  permit  development  of  operating  instructions  for  test  participants  as 
called  for  in  Section  2.4.5,  "Orientation  Plan." 

2. 4.6, 2  Site-Supplied  Materials. 

Test  participants  are  required  to  provide  their  own  terminal  with  screen 
and  keyboard,  modem,  and  local  telephone  line  connection.  Terminals  used  for 


the  benchmarking  should,  as  a  minimum,  be  capable  of  receiving  and  trans¬ 
mitting  the  full  American  Standard  Code  for  Information  Interchange  (ASCII) 
character  set.  The  screen  used  should  provide  a  full  80-columns  by  24-lines 


display.  It  is  recommended  that  the  modem  and  teleprocessing  features  of  the 
terminal  provide  the  following  features: 


1200-baud  data  transmission  rates; 

-  full  duplex; 

-  asynchronous  communications;  and 

-  parity  checking  (odd  or  even  parity) . 

Test  participants  will  connect  to  either  the  host  computer  used  at  LLNL  for 
benchmarking  or  the  host  computer  designated  by  the  vendor  for  those  packages 
not  installed  at  LLNL. 

At  the  conclusion  of  benchmarking,  participants  must  submit  a  written 
evaluation  of  the  packages  using  the  checklists  provided  in  the  Test  Plan 
(Appendices  A  through  H) .  These  evaluations  will  be  compiled  by  LMI  for  use 
in  selecting  a  package  for  prototype  implementation. 

2.4.7  Security  and  Privacy. 

Test  catalogs  extracted  from  the  DTIC  TR  data  base  and  the  DNA  ATLAS  file 
will  consist  of  citations  that  are  unclassified  with  unlimited  and 
unrestricted  distribution.  However,  catalogs  created  at  vendor  sites  from 
these  extracts  must  be  safeguarded  through  a  nondisclosure  agreement  with  the 
vendor.  No  data  subject  to  Privacy  Act  restrictions  will  be  used:  test 
patron  data  (if  required)  can  be  fictitious.  Once  implemented  at  an  opera¬ 
tional  site  for  testing,  the  catalogs  and  patron  file  will  be  subject  to 
protection  and  access  limitations  consistent  with  the  level  of  classification 
of  the  citations  present  in  the  catalog.  WARNING :  During  benchmarking, 
participants  may  be  required  to  enter  bibliographic  data  to  test  cataloging 


features.  Do  not  enter  classified  data  in  any  of  the  benchmark  systems . 
Limit  all  cataloging  entries  to  unclassified,  unlimited  distribution  citations 


2.5  Phase  II  and  III. 


Gateway  Integration.  Phase  II  benchmarking  encompasses  integration  of 
the  intelligent  gateway  software  with  the  commercial  package  selected  during 
Phase  I  testing.  Phase  III  provides  the  opportunity  to  test  and  refine  the 
integration  of  the  two  software  components  prior  to  prototype  system 
implementation  at  the  DNA. 

User  Interface  Implementation  and  Refinement.  To  meet  the  objectives  of 
the  system  design,  users  must  have  access  to  the  system  functions  through  a 
common  set  of  commands  and  formats.  Prior  to  prototype  system  implementation, 
a  user  command  language  and  format  will  be  installed  and  refined  in  response 
to  user  requirements.  Initial  requirements  for  the  user  interface  will  be 
described  in  a  Program  Specification. 


SECTION  3.  TEST  SPECIFICATION  AND  EVALUATION 


3.1  Test  Specification. 

3.1.1  Requirements . 

Functions  and  processes  required  for  the  LAM  system  have  been  documented 
in  the  Functional  Description  and  System  Specification.  These  requirements 
are  reflected  in  the  benchmarking  checklists  contained  in  Appendices  A 
through  H  of  this  Test  Plan.  The  benchmarking  checklists  serve  as  score 
sheets  and  are  distributed  to  test  participants  as  part  of  the  Test  Plan.  Use 
of  the  checklists  and  score  sheet  is  described  in  Section  4  of  this  Test  Plan. 
Separate  benchmarking  checklists  are  provided  for  each  major  system  function: 

-  Reference  —  Appendix  A; 

-  Cataloging  —  Appendix  B; 

-  Circulation  Management  and  Control  —  Appendix  C; 

-  Reports  and  Other  Printed  Output  —  Appendix  D; 

-  Security  and  Access  Control  —  Appendix  E; 

-  Reliability  and  Maintainability  —  Appendix  F; 

-  Ease  of  Use  and  Modification  —  Appendix  G;  and 

-  Hardware  and  Software  Features  —  Appendix  H. 

3.1.2  System  Functions. 

Phase  I  benchmarking  will  focus  on  verifying  the  availability  of 
functions  supporting  cataloging,  retrieval,  and  circulation  management  and 
control.  The  scope  of  Phase  I  benchmarking  is  limited  to  assessments  of 
commercial  software  packages  operated  independently  of  intelligent  gateway 
software.  For  this  reason,  benchmarking  will  be  confined  to  cataloging  into  a 
local  catalog  and  retrieving  cataloged  citations  from  a  local  catalog. 
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Collection  management  features  supporting  circulation  management  and  control 
will  also  be  benchmarked.  A  detailed  discussion  of  benchmarking  procedures 
and  a  listing  of  features  to  be  covered  are  contained  in  Section  4  and 
Appendices  A  through  H,  respectively. 

There  are  over  350  features  included  for  evaluation  in  the  Test  Plan. 
While  some  of  these  features  may  not  appear  essential  for  prototype  system 
implementation,  they  have  all  been  included  to  make  the  benchmarking  as  com¬ 
prehensive  and  complete  as  possible.  To  aid  test  participants  in  identifying 
essential  or  critical  features ,  the  benchmarking  checklists  included  in 
Appendices  A  through  H  have  been  annotated  with  "  *  "  in  the  left-hand  margin 
for  these  features. 

Some  features  marked  with  an  "  *  "  may  appear  different  from  or  contra¬ 
dict  features  not  marked  as  critical.  This  apparent  contradiction  occurs 
because  checklists  allow  for  differences  in  implementation  methods  among 
software  packages.  Where  alternative  implementation  methods  are  most  likely 
to  be  used  to  meet  requirements  for  critical  features,  the  alternative  methods 
are  listed  for  evaluation.  By  doing  this,  the  evaluation  can  be  more  compre¬ 
hensive  and  may  differentiate  a  superior  product  from  among  the  candidate 
systems. 

3.1.3  Test/Function  Relationships. 

The  test  elements  for  each  function  subject  to  evaluation  are  listed  in 
the  appendices.  Section  4  presents  a  description  of  the  benchmarking  check¬ 
lists  contained  in  the  appendices. 

3.2  Test  Methods  and  Constraints. 

3.2.1  System  Test  Conditions. 


Each  package  will  be  benchmarked  using  dial-up  terminal  access  to  a 
computer  running  the  package.  A  test  catalog  of  approximately  1,000  records 


(see  Section  4.2.2,  "Test  Data,"  for  a  description  of  the  test  catalog)  will 
be  used  to  evaluate  retrieval  functions.  When  evaluating  cataloging  func¬ 
tions,  test  participants  will  create  citations  that  will  be  added  to  this 
catalog  by  the  update  features  offered  by  the  tested  package.  Test  partici¬ 
pants  are  again  cautioned  not  to  create  classified  or  limited  distribution 
citations  during  the  test.  Inputs  used  by  test  participants  for  retrieval  and 
cataloging  of  citations  will  be  those  specified  for  the  package  being  tested. 

3.2.2  Extent  of  System  Test. 

Each  of  the  six  packages  listed  in  Section  2.2,  "Pretest  Activity,"  will 
be  evaluated  for  the  availability  and  relative  performance  of  those  functions 
and  features  related  to  cataloging,  retrieval,  and  circulation  management  and 
control.  These  functions  encompass  the  full  set  of  capabilities  specified  for 
the  prototype  system.  Section  4  and  the  appendices  delineate  the  specific 
functions  and  features  included  in  the  evaluation. 

3.2.3  Data  Recording. 

Test  participants  are  required  to  use  the  checklists  and  score  sheets 
included  as  part  of  this  Test  Plan.  One  complete  set  of  benchmarking  check¬ 
lists  must  be  completed  for  each  package  subjected  to  benchmarking.  Test 
participants  may  make  additional  copies  of  the  benchmarking  checklists  con¬ 
tained  in  the  appendices  for  the  purpose  of  conducting  the  evaluations 
prescribed  in  this  Test  Plan.  Test  participants  are  strongly  encouraged  to 
make  additional  written  notes  and  observations  using  separate  sheets  if  room 
on  the  checklists  is  not  sufficient.  An  explanation  on  how  to  use  the 
checklists  is  included  in  Section  4. 

3.2.4  System  Test  Constraints. 


Dial-up  Access.  Some  packages  may  make  use  of  special  features  that  use 
cursor  control  commands  unique  to  a  particular  type  of  remote  terminal.  Many 


of  these  features  may  be  unavailable  for  evaluation  because  test  participants 
will  be  using  dial-up,  asynchronous  access  methods. 

Availability  of  Features.  In  some  cases,  the  packages  being  benchmarked 
may  not  offer  a  feature  listed  on  the  checklist.  In  addition,  some  features 
may  be  available  in  a  package  but  not  be  available  for  evaluation  because  of 
lack  of  direct  access  to  the  applications  software  or  computer  operating 
system.  If  test  participants  encounter  circumstances  in  which  these  condi¬ 
tions  exist,  they  should  note  the  feature  and  the  circumstances  on  the  check¬ 
list.  Periodically  —  probably  biweekly  —  throughout  the  benchmarking 
period,  test  participants  will  be  contacted  to  discuss  any  problems  arising 
during  the  benchmarking.  When  contacted,  they  should  indicate  the  features 
that  they  are  unable  to  evaluate  and  provide  a  brief  explanation  of  why  those 
features  cannot  be  evaluated. 

Test  Catalog.  The  test  catalog  implemented  for  benchmarking  will  contain 
approximately  1,000  citations.  This  catalog  is  small  compared  with  the 
expected  size  of  a  typical  technical  library  catalog.  As  a  result,  retrieval 
times  experienced  during  the  benchmark  may  not  be  representative  of  actual 
system  operations. 

Multiple  Users.  Over  the  entire  time  of  the  benchmark,  it  may  be  diffi¬ 
cult  to  simulate  the  number  of  simultaneous  users  expected  on  the  prototype 
system.  Thus,  better  system  response  times  may  be  noted  during  benchmarking 
than  would  be  expected  with  an  operational  system  supporting  several  users 
simultaneously. 

Schedule .  The  software  packages  provided  on  vendor  computers  will  be 
available  for  testing  during  periods  specified  by  the  vendor.  The  schedules 
established  by  each  vendor  will  be  announced  to  test  participants  through  sup¬ 
plemental  notices  published  as  the  pretest  system  implementation  progresses. 


Test  participants  may  elect  to  evaluate  system  functions  in  any  order. 
However,  the  benchmarking  checklists  list  processing  steps  and  features  by 
function  in  the  sequence  or  order  in  which  they  would  be  executed  during 
normal  operations.  Therefore,  test  participants  are  strongly  encouraged  to 
move  through  the  sequence  of  test  steps  within  a  function  as  listed  on  the 
accompanying  checklists  (see  Appendices  A  through  H) . 

3.4  Test  Evaluation. 

3.4.1  Test  Data  Criteria. 

Results  will  be  evaluated  using  the  relative  performance  scores  assigned 
to  each  group  of  processes  or  functions  by  the  test  participants.  These 
relative  rankings  will  be  combined  to  form  an  overall  ranking  for  each  system. 
In  general,  ranking  preference  will  be  given  to  systems  offering  the  full  list 
of  features  and  capabilities  specified  as  LAM  system  requirements.  Additional 
weighting  will  be  given  to  features  established  as  critical  for  system 
implementation.  Since  each  vendor  may  choose  to  implement  those  features  and 
capabilities  differently,  a  premium  will  be  placed  on  user  reaction  to  the 
systems  as  reflected  in  the  relative  rankings  provided  by  test  participants. 

3.4.2  Test  Data  Reduction. 

Test  participants  must  record  their  evaluation  of  system  features  using 
the  checklists  provided  with  the  Test  Plan.  Data  on  the  checklists  will  be 
combined  to  form  an  overall  ranking  for  the  systems.  System  rankings  devel¬ 
oped  by  individual  test  participants  will  count  equally  among  all  participants 
when  combined  to  form  an  overall  ranking.  The  overall  rankings  will  not 
reflect  weighting  of  one  function  over  another.  In  the  event  of  ties  or 
ambiguities  in  the  ranking  results,  a  structured  walk-through  of  the  test 
sequence  may  be  repeated  using  the  packages  in  question.  Selected  test 


SECTION  4.  PHASE  I  TEST  DESCRIPTION 


4.1  Test  Description. 

In  the  Phase  I  benchmarking,  the  test  participants  will  prepare  and 
submit  evaluations  of  the  six  software  packages  chosen  for  benchmarking. 
These  evaluations  will  be  used  to  determine  which  of  the  packages  are  best 
suited  for  use  in  the  LAM  prototype  system.  The  checklists  in  Appendices  A 
through  H  will  be  used  by  the  test  participants  for  this  purpose.  Each  of  the 
eight  checklists  encompasses  either  a  library  function,  such  as  "Reference" 
(Appendix  A),  or  a  hardware/software  function,  such  as  "Reliability/ 
Maintainability"  (Appendix  F).  While  some  duplication  does  exist  among  the 
checklists,  they  are  self-contained  and  can  be  used  independently  of  each 
other.  The  checklists,  contained  in  the  appendices  of  this  Test  Plan,  cover 
the  following  functions  and  operating  features: 

-  Reference  —  Appendix  A; 

Cataloging  —  Appendix  B; 

-  Circulation  Management  and  Control  —  Appendix  C; 

-  Reports  and  Other  Printed  Output  —  Appendix  D; 

-  Security  and  Access  Control  —  Appendix  E; 

-  Reliability  and  Maintainability  —  Appendix  F; 

Ease  of  Use  and  Modification  —  Appendix  G;  and 

-  Hardware  and  Software  Features  --  Appendix  H. 

Each  checklist  is  comprised  of  software  features  grouped  into  categories. 
Thus,  a  three-tiered  hierarchy  has  been  created:  functions  comprised  of 
categories  comprised  of  features.  The  packages  will  be  tested  by  feature; 
that  is,  test  participants  will  conduct  the  test  by  determining  whether  each 


package  offers  the  features  on  the  checklist.  Examples  of  the  features  are: 
"System  provides  on-line  updating  of  catalog,"  "System  provides  global  search 
and  replace,"  and  "System  provides  record  locking  for  update." 

The  features  on  the  checklist  have  been  constructed  to  be  answered  in  a 
"yes/no"  format  with  scoring  levels  designed  to  differentiate  among  the  per¬ 
formance  levels  of  the  six  packages  tested.  This  format  is  easy  to  use  and 
minimizes  differences  in  judgment  among  the  test  participants. 

Because  some  software  packages  may  offer  features  in  a  slightly  different 
form  than  required  in  the  LAM  or  in  a  form  that  the  test  participants  may  find 
difficult  to  use,  a  five-point  scoring  has  been  used.  This  scoring  system 
will  allow  test  participants  to  indicate  not  only  that  a  feature  is  or  is  not 
available  but  also  that  a  feature  is  available  with  limitations.  The  scale  to 
be  used  by  test  participants  in  rating  each  feature  is  as  follows: 

-  1  —  No,  the  packages  does  not  offer  the  feature; 

-  2  —  Yes,  the  package  offers  the  feature  but  with  severe  limitations; 

-  3  —  Yes,  the  package  offers  the  feature  with  moderate  limitations; 

-  4  —  Yes,  the  package  offers  the  feature  with  minor  limitations;  and 

-  5  —  Yes,  the  package  offers  the  feature  without  limitations. 

To  complete  the  benchmarking  checklist,  a  test  participant  exercises  the 
features  listed  for  the  package  being  evaluated,  checks  the  appropriate  score 
column  (1,  2,  3,  4,  or  5),  and  adds  any  comments  in  the  column  specified  on 
the  checklist.  Separate  sets  of  checklists  must  be  filled  out  for  each  system 
tested.  Test  participants  are  authorized  to  duplicate  the  checklists  con¬ 
tained  in  the  appendices  for  use  during  the  evaluation. 

To  utilize  the  particular  expertise  of  the  various  test  participants 
and  to  minimize  the  time  required  by  any  participant,  we  recommend  that 


primary  responsibility  for  completing  the  checklists  be 
follows : 


assigned  as 


-  Defense  Nuclear  Agency  (DNA) : 

--  Appendix  A  --  Reference, 

—  Appendix  B —  Cataloging, 

—  Appendix  C  —  Circulation  Management  and  Control,  and 
--  Appendix  D  --  Reports  and  Other  Printed  Output; 

-  Lawrence  Livermore  National  Laboratory  (LLNL) : 

~  Appendix  E  —  Security  and  Access  Control, 

--  Appendix  F  --  Reliability  and  Maintainability,  and 
—  Appendix  H  --  Hardware  and  Software  Features; 

-  Shared  Responsibility  (DNA  and  LLNL) : 

—  Appendix  G  —  Ease  of  Use  and  Modification. 

Staff  members  from  DTIC  will  also  complete  checklists  for  Reference 
(Appendix  A),  Cataloging  (Appendix  B),  and  Reports  and  Other  Printed  Output 
(Appendix  D) . 

While  responsibilities  for  completing  each  of  the  benchmarking  checklists 
are  assigned  to  specific  test  participants,  all  participants  are  invited  to 
prepare  and  submit  evaluations  on  any  of  the  eight  functions. 

4.2  Test  Control. 

4.2.1  System  Test  Means  of  Control. 

The  test  participants  will  initiate  and  run  all  tests  at  a  time  of  their 
choosing.  Neither  the  initiation  nor  the  pace  of  the  benchmarking  is  pre¬ 
determined.  Control  of  the  benchmarking  is  maintained  by  the  individual  test 
participant.  An  exception  to  this  may  arise  for  software  running  on  systems 
not  resident  at  LLNL.  For  that  software,  the  participants  will  need  to  con¬ 
duct  evaluations  during  the  operational  hours  of  the  host  system.  Specific 


schedules  for  using  those  systems  will  be  provided  to  test  participants  as 
supplemental  instructions  to  the  Test  Plan.  Participants  must  complete  their 
evaluations  by  the  benchmarking  termination  date  shown  in  Figure  2-1,  "Mile¬ 


stone  Chart  for  Phase  I  Benchmarking." 
4.2.2  Test  Data. 


The  data  to  be  used  in  this  benchmark  consist  of  500  records  extracted 
from  the  DROLS  and  500  records  extracted  from  the  DNA  ATLAS  file.  All  records 
are  unclassified  and  unrestricted.  Appendix  I  contains  a  description  of  the 
citation  format  extracted  from  DROLS.  Similarly,  Appendix  J  contains  a  brief 
description  of  the  citation  format  for  the  DNA  ATLAS  extract. 

4.3  Test  Procedures. 

This  section  provides  the  recommended  procedures  for  conducting  the 
benchmarking. 

4.3.1  Test  Setup. 

LLNL  is  responsible  for  loading  the  six  software  packages  to  be 
benchmarked  and  assuring  that  they  are  available  via  dial-up  terminals  to  the 
other  test  participants.  LLNL  is  also  responsible  for  preparing  material  that 
summarizes  the  commands  for  operating  each  package  and  for  distributing  that 
material  to  each  of  the  test  participants  listed  in  Section  2.4.4, 
"Personnel. " 

When  the  test  participants  receive  their  material,  they  are  encouraged  to 
log-on  to  each  system  and  gain  some  familiarity  with  the  six  packages  before 
beginning  the  formal  evaluation.  They  are  especially  encouraged  to  learn  to 
use  the  "Help"  feature  of  each  package  to  enable  them  to  work  out  their  own 
difficulties  during  the  evaluation. 

4.3.2  Test  Initialization. 

Each  test  is  initialized  by  logging-on  to  the  system  running  the  package 
to  be  evaluated,  using  the  material  provided  by  LLNL. 


4.3.3  Test  Steps. 

To  perform  the  tests,  a  participant  should  first  choose  the  function  to 
be  evaluated.  Test  participants  are  encouraged  to  read  through  each  bench¬ 
marking  checklist  completely  before  performing  any  evaluations.  Furthermore, 
we  recomnend  that  participants  complete  the  evaluation  of  a  function  for  all 
six  packages  before  evaluating  any  other  functions.  This  procedure  will 
enable  test  participants  to  make  assessments  on  the  relative  performance  of 
each  package  in  determining  scores  for  each  feature. 

While  the  benchmarking  procedures  do  not  require  participants  to  comment 
on  the  performance  of  the  packages  (beyond  assigning  a  numerical  score), 
comments  will  be  useful  in  cases  in  which  numerical  scores  do  not  provide 
sufficient  differentiation  among  the  systems. 

After  choosing  the  function  to  be  evaluated,  the  participants  should 
evaluate  each  of  the  packages  for  that  function.  This  evaluation  involves 
rating  the  package  for  all  the  features  on  the  checklist  using  the  five-point 
scale  shown  in  Section  4.1,  "Test  Description."  As  stated  previously,  while 
the  participants  are  required  only  to  assign  a  numerical  score  for  each 
feature,  they  are  encouraged  to  provide  written  comments  whenever  possible  to 
assist  in  the  overall  evaluation. 

4.3-4  Test  Termination. 

The  evaluation  is  complete  when  the  participants  have  rated  each  feature 
of  each  package  for  the  functions  within  their  area  of  responsibility  (see 
Section  4.1,  "Test  Description,"  for  recommended  assignments).  When  the 


checklists  are  complete,  they  should  be  sent  to  the  Logistics  Management 
Institute,  6400  Goldsboro  Road,  Bethesda,  Maryland  20817-5886,  Attention: 
DL501  Project  Leader. 


APPENDIX  A 


BENCHMARKING  CHECKLIST:  REFERENCE 

This  appendix  contains  the  benchmarking  checklist  for  the  "Reference" 
function  and  features.  One  set  of  checklists  must  be  completed  for  each 
different  system  tested.  Check  only  one  score  per  feature.  Indicate  your 
score  by  placing  a  mark  in  the  column  corresponding  to  the  score  you  feel 
most  accurately  describes  the  feature.  Use  the  following  scoring  scale  and 
corresponding  criteria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 
side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 

Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 

using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  pay  particular  attention  to  the  features 
marked  with  "  *  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 

Leader,  6400  Goldsboro  Road,  Bethesda,  MD  20817-5886.  Completed  checklists 
should  be  mailed  to  this  address  also. 


A- 1 


PREVIOUS  PAGE 
IS  BLANK 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


« 

P  iH 
•H  O 

5  a 

r-t  n 

a 

X  .. 

©  o  — > 

(4 

no© 
p  ©  n 
(.9  9 

o  — 

o.  >1 

a  e  a 
9  o 
O  H  TJ 
P  0 

B  ©  p 
©  o  (. 

■PC© 

n  9  n 
>»  t.  c 
co  -p  n 


<W  t. 

o  « 

n  (.  ■§ 
c  ©  9 
o  jq  a 
•h  a 

■PC® 
It  B  H 
X  X  C  A 

X  -H  rl  rl  U  © 

•H  <h  <m  A  ©  ■H 

©  <n  a  X  C. 

2  C.  3  O  tH  © 

•H  Q.  ©  O  ©4  > 


I  I  I  I  I  I 


a  ts 

>>  ©  p 

p  a.  c 
>»  © 
p  p  •o 
C  -H 

©  T3 
Bh  © 

«  ©  a 

r4  «H  © 

©  V.  .9 

(. 

©  ©  ©  © 
p  p  p  jg 
©  ©  ©  P 
T3  TJ  T3  O 


I  I  I  I  I  I 


I  I  I 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


p  o 
C  X 
®  « 

o 

«  « 
t,  o  .c 
u  o  c 
■pot.® 

n  o  «  J3 

§®  ®  p 
t.  n  o 


a. 

o 

p  -p 

n  a 

P 

p  a 

« 

•E  3  ^ 

O  0.  St 

*  a  a 

<o  t, 

a  a 

o  >»  © 

P  r-(  « 

•  -a  t. 

(0  O  ® 

p  w 
w  p  3 

>>  « 

o  a  o 

r-t  OP® 

a  p  c 

B  3  P  O 

®  ®  m  p 

p  « 

B  BP® 

®  ®  ® 

p  p  -a  t, 

n  n  t.  a. 

>.  >»  O  X 

CO  CO  9  o 


n 

n  n  ® 

•OP® 

c  «  c 
®  ®  o 

#3  0. 

iff® 
o  ®  ® 
O  C.  u 

S3  S3  S3 
o  o  o 

fy  k  s 


t,  « 

o  n 
o 
n  t, 
a  o, 
tk  x 
o  ® 

Ch 

■X  S3 
U  O 
O  U 
*  « 
o 
®  a 
® 

o  t> 
p  o 
>  <p  c 
o  o 

t,  m  p 

a  «  p 
p  at 
B  «  p 
®  P  3 
P  Q.  B 

a  B  c. 
>»  ®  o 

CO  P  t 


fixed  menu  approach 
flexible  menu  approach 
user  selection  of  menu  level 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


*  > 


s? 


Em 


© 

JB 

+> 


C 

O 


S  9 


toe  a 

c  c  -H 

■H  3  +> 
X3  b 
O  w  0 

i  ra  o 
0  0 
n  n  +» 
(0  © 

© 

+>  — > 

(.<0  0 
o  +>  c 

O*  CD  *H 

a-o  js 
3  o 
n  ©  © 
r-t  a 

a  a 

©•Hi—) 

+>  +>  <d 
B  H  O 
>*3  0 

M  S  rl 


as 


+> 

3 

a* 

c 


c 

o 


a 

a 

as 

b 

a. 

X 

© 


A 

O 

b 

© 


4 


b 

« 

•O 

as 

© 

b 


b 

© 

4-> 

© 

% 

JS 

o 


•3 


*2 
© 

o  o 

A  3  <H 
>»  C  4-> 

«  ©  a 
j*  S  o 


i  i  i 


3 


> 

© 


CM 


© 

3 

a 


n 

a 


© 

4-> 

© 

•a 


T> 

© 

c 


ss 
o 
u 
© 
a  © 
o  © 

b 

b.  9 


b* 

o 


s* 


a 

+>  •• 
b  n 
o  +» 
a.  a 
a  •h 

3  O 
a  a. 


n 
n  b 

&  ° 
O  J3 
£  +> 
*»  3 
3  © 
© 

© 


b. 

© 

■O 


a  © 

9* 


©  fep 

■o  c 

b  -H 
O  © 
O 

©  jC 

b  +> 


b 

© 


B  a 
©  a 
+>  © 
n  o 
>»  o 
to  © 


S© 
b 

0  o  o 

rH  ©  Q. 
+>  b  b 
■H  ©  O 
+>0.0 


a 
■o  © 

«  o  « 

©  «H 

eve 
«h  © 
+i  «  > 
O  •H  ® 
©  b  ■H 
■>1  b 
A  >»+> 


a  3 
© 

>  n 
©  © 


b  i-» 
*>  b< 
© 

b  © 


3  C  « 
R  ©  b 


I  I  I  I  I 


a  q. 

©  ■H 
+>  +> 
©  .m 

£8 


A 
<d  r-4 

©  2 
©  +> 
•o  c 

•H  O 
>  O 
O 

b  H 

a  © 
> 

a  © 

©  -h 
+»  b 
©  +> 
>*  © 
to  b 


A- 6 


*  • »  V-  ■ .  * .  • 


.  -  ---  .‘“.v 

*  .V 


?  •  t*-*  m  •  •  '  b  '  k  *  fc  •  *■.+•**"*•*»  *  ’  i  * 

••  ■•‘•y v-v .■ v*v>: 


set/change  defaults  for  display 
formats 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


"O 

0) 

®  > 
n  at 

XS-H  B 

4-1  n 
tO  *h 

c  c  o 
«  ® 
rH  ®  ® 
t4  a 
©  O  -H 


a  s* 
®  o 
x: 
■H  *1 
3 

to  to 

4-1 

to  £4 
BOO 


x  a 
®  u 

■S  §J2 

•H 

■G  JS 
Oh  4-1  O 
O  H  £. 

s  to 
>»  ® 
B  ^  B 

0.0*0 
no.® 
rt  a  4) 

TO  ®  tO 


o  c 

a  4-i 

n 

« 

£4 

0 

£4  rH 

® 

0 

3 

n 

rH  *0 

n 

n  ® 

a  £. 

c 

<0  rH 

£.  Oh 

4-1 

rH  V 

3 

4-1  n 

4J  O 

3 

® 

O  rH 

4-1 

n  c 

£4 

<M  > 

£4 

£4 

£4  O 

£4 

ra  4J 

rH 

to 

4-1  O 

0 

V 

3 

0  c 

O 

4-1 

c 

® 

a 

rH  *H 

Q. 

X  *H 

tO 

T3 

Q.  ® 

a.  js 

>*  rH 

M  >> 

a  4-1 

a 

®  £4 

n 

£ 

a  x: 

0.  0 

>> 

to  e. 

tO  %H 

JS 

0 

•H  to 

3 

•O  *1 

®  £< 

to 

3  * 

3  *ri 

rH 

rH  Q. 

Q.  -H 

o 

<M 

rH  N 

n 

e  ® 

J3  * 

» 

n 

n  -C 

44 

O.  ® 

O 

£4 

H 

-rl  £4 

4-1  9 

x 

>* 

» 

rH 

n  a  4J 

4-1  « 

to 

4-1 

a  £4 

a 

U 

0 

B  « 

a 

O 

•H  "H  ® 

®  a 

« 

® 

V  0 

« 

c  e 

a  0 

as 

®  rH 

®  m 

rH 

O  *)  B 

a  n 

n 

n 

4-1  x: 

4-> 

rH  -rl 

rH  Cm 

X5 

4-10.® 

4-1  a 

rH 

n  +> 

n 

B  B  rl 

n  £4 

a 

>>  3 

>* 

1  1 

1  1 

1 

>1  rl  rl 

>>  ® 

X 

1  1 

i  i 

1 

CO  (0 

CO 

CO  "O  £h 

CO  4-1 

V 

System  supports 
qualifiers 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


o 

p 

® 

v 

a  o  - 
O  V.  c 
u  o 

Vi  A  P 
3  P 
H  ^  flJ 
#  3  H 
>(0  3 

«  «  S 

P  ®  V 
O 

P  P  Vt 
® 

V  V  C 
O  O 


A 

® 

O 

®  3 

60  T3 

A  O 

a  v 

a  a 

S 

0  p 


p  ® 

A 

A 

A  ® 

V  P 

A 

P 

V  Vi  P 

o  P 

® 

P 

® 

®  o  a 

a  V. 

V 

a  •• 

A 

P  P 

Qi 

a. 

u  >» 

V 

Vi  V 

3  >i 

X 

®  S> 

® 

POP 

A  P 

® 

Q. 

> 

P  A  A 

P 

■a 

P 

A  B 

B  V 

St 

a  ® 

c 

3  3V 

®  o 

o 

®  p 

3 

cr  a  o 

P  33 

V 

p  p 

A  P 

A 

a  a 

>*  3 

® 

>»p 

1 

1  1 

CO  A 

A 

CO  P 

®  a 

o  « 

3  a 

S  § 

v  a 

a.  A 

®  -o 

P  v  ® 

a  i-'  P 

>  A 

®  AO 

■h  pc 

V  P  3 

P  33  V 

®  P 

V  Vi 

O  Vi 

a  o 

o  P 

P  CP 

P  3  A 

A  OP 

P  O  rH  A 

P  C 

O  P  P  O 

I  (DAP 

®  P  p  P 

p  p  P  A 

a,  p  p  P 

p  ccp 

P  p  p  o 


i  i  i 


i  i  i 


-  y.v/rV-.*- 


subsequent  list  of  expanded 
partial  citations  on  reques 
full  citation  on  request 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


T3 

V 

■U 

(0 

o 

m  a 
was 

st,  t. 

©  © 


*>  a 

a 

o.  n 

-<->  >. 

3  <0 

O  r- » 

a 

•a  n 

©  *H 

4J  T3 

a 

•h  a 

u  © 

PL.  *> 

v.  n 

>»  >» 

©  CO 


c 

o 

Cm  -H 

4-> 

O  © 

S0  -H 
O  4J 
.H  -H  C 
C  O 
'O  "H  "H 
C  4-> 

©  c  © 
o  c 
a  -h  Ti 

o  n  m 
eon  b 
o  v  « 
h  n  4J 


ii  iiii 


i  i  i  i  i 


budget  and  other  financial 
data  displays 
routing  data  displays  (for 
serials) 

delinquency  data  displays 


w 

>»  ■o 
mac 
B  h  a 
©  a 
■u  co  <n 

•H  -H  4-1 

•a  cd 


Li  w  •• 

©  >) 
jb  a  « 

44  <D  I— I 

O  4-1  Q. 

n  n 

>»  *H 
I  CO  TJ 


T3 

a 

A 

s 

Cm 

c 

© 

>> 

k. 

rH 

© 

J* 

© 

> 

Cm 

o 

a 

A  W 

Cm 

© 

4-> 

Cm 

w 

3 

o 

A 

“H 

a 

•H 

C 

» 

-k> 

Cm 

© 

•o 

T3 

■rl  Cm 

A 

4-> 

© 

O 

w 

T3 

© 

■o 

4-> 

T3 

•a 

© 

k. 

© 

o 

Cm 

©  © 

•o 

Cm 

4-> 

V 

O 

>>  a 

•H 

O 

Cm 

A 

rH 

r-^ 

O 

A  >» 

> 

Cm 

O 

n 

© 

4-1 

© 

rH  4-1 

o 

> 

w 

c 

k. 

a 

k. 

A 

4-1 

© 

1 

© 

W  £3 

a 

© 

a 

k. 

k, 

4-1 

rH 

•H  O 

4-1 

3 

a 

V 

c 

rH 

T3  A 

a 

A 

o 

a 

w 

o 

3 

V 

© 

rH 

o 

A 

3 

o 

Cm 

A 

4-1 

a 

44  k. 

W 

a 

A  O 

>1 

« 

1 

i 

1 

Q  Cm 

CO 

44 

CD  k. 

rH  O 

a  a 

k,  c. 

o  o 

Cm  Cm 

-X 

S  k, 

o  o 

H  * 

w 

n  o 

03  -tH 

U  JZ 

a  a. 

x  a 

0)  I)  k 

>»  to 

£  Si  O 
O  SS  ■H 
Cm  ©  rH 
A  4-1  £3 

©  A  -h 

n  to  .o 


0)  O 
w  +1 

at  <d 
-*-3  w 


A 

>> 

«k 

4-> 

T3 

A 

w 

>» 

CQ 

t — 1 

c 

44 

•H 

W 

a 

o 

« 

H 

3 

w 

•H 

O 

•rt 

44 

Cm 

• 

•rH 

n 

A 

A 

00 

t} 

Cm 

n 

> 

• 

<d 

A 

c 

> 

<D 

> 

•H 

© 

A 

'w* 

k. 

.a 

W 

W 

a 

W 

0) 

cd 

44 

C 

-O 

« 

4-> 

Cm 

o 

A 

X} 

cd 

o 

•H 

■H 

S 

* 

<d  4-> 

H  k  k 

Q.  O  © 

a  a£ 

©  ©  44 

*>  k  O 


Q.  4.)  - 

Q.  ffl  (0 

3  *)  41 

n  c  « 
©  -a 
a  n 
©  ©  -o 

4-1  L.  © 

wan 
>»  ©  o 

CO  k  o 


>  k.  T3 
O  O  © 

k  Cm  £ 

a  a 

>•  A 
8  «  U 
©  rH  tO 
4->  a  A 
W  W  C, 
>>  -H  Cd 

co  n  a 


BENCHMARKING  CHECKLIST:  REFERENCE  (CONTINUED) 


System  permits  user  specification 
of  system  device  to  be  used  for 


APPENDIX  B 


BENCHMARKING  CHECKLIST:  CATALOGING 

This  appendix  contains  the  benchmarking  checklist  for  the  "Cataloging" 
function  and  features.  One  set  of  checklists  must  be  completed  for  each 
different  system  tested.  Check  only  one  score  per  feature.  Indicate  your 
score  by  placing  a  mark  in  the  column  corresponding  to  the  score  you  feel 
most  accurately  describes  the  feature.  Dse  the  following  scoring  scale  and 
corresponding  criteria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 
side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 
Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 
using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  pay  particular  attention  to  the  features 
marked  with  "  *  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 
Leader,  6400  Goldsboro  Road,  Bethesda,  MD  20817-5886.  Completed  checklists 
should  be  mailed  to  this  address  also. 
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System  provides  convenient 
workforms  or  templates  for  entry  of 
bibliographic  data  during 
cataloging 
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During  on-line  cataloging,  the 
system  allows  the  user  to  edit  and 
correct  input  data  without  re¬ 
entering  the  entire  record 
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System  automatically  restricts  or 
oontrols  user  access  for  modifying 
or  updating  data  base  (catalog) 
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APPENDIX  C 


BENCHMARKING  CHECKLIST:  CIRCULATION  MANAGEMENT  AND  CONTROL 

This  appendix  contains  the  benchmarking  checklist  for  the  "Circulation 
Management  and  Control"  function  and  features.  One  set  of  checklists  must  be 
completed  for  each  different  system  tested.  Check  only  one  score  per  feature. 
Indicate  your  score  by  placing  a  mark  in  the  column  corresponding  to  the  score 
you  feel  most  accurately  describes  the  feature.  Use  the  following  scoring 
scale  and  corresponding  criteria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes ,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 
side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 
Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 
using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  f'iy  particular  attention  to  the  features 
marked  with  "  •  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 

Leader,  6400  Goldsboro  Road,  Betbesda,  MD  20817-5886.  Completed  checklists 
should  be  mailed  to  this  address  also. 
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IINs  are  usable  in  other  library 
applications  such  as  serials 
control  and  circulation  control 
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System  permits  recall  of  items  on 
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In  printed  form 
on-line 
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System  records  statistics  for 
reserve  collection  activity 
separately  from  those  for  regular 
collection 
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APPENDIX  D 


BENCHMARKING  CHECKLIST:  REPORTS  AND  OTHER  PRINTED  OUTPUT 


This  appendix  contains  the  benchmarking  checklist  for  "Reports  and  Other 

Printed  Output"  features.  One  set  of  checklists  must  be  completed  for  each 

different  system  tested.  Check  only  one  score  per  feature.  Indicate  your 

score  by  placing  a  mark  in  the  column  corresponding  to  the  score  you  feel 

most  accurately  describes  the  feature.  Use  the  following  scoring  scale  and 
corresponding  criteria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 
side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 

Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 

using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  pay  particular  attention  to  the  features 
marked  with  "  *  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 

Leader,  6400  Goldsboro  Road,  Bethesda,  MD  20817-5886.  Completed  checklists 
should  be  mailed  to  this  address  also. 
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BENCHMARKING  CHECKLIST:  SECURITY  AND  ACCESS  CONTROL 


This  appendix  contains  the  benchmarking  checklist  for  "Security  and 
Access  Control"  features.  One  set  of  checklists  must  be  completed  for  each 
different  system  tested.  Check  only  one  score  per  feature.  Indicate  your 
score  by  placing  a  mark  in  the  column  corresponding  to  the  score  you  feel 
most  accurately  describes  the  feature.  Use  the  following  scoring  scale  and 
corresponding  criteria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 
side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 
Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 
using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  pay  particular  attention  to  the  features 
marked  with  "  *  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 

Leader,  6400  Goldsboro  Road,  Bethesda,  MD  20817-5886.  Completed  checklists 
should  be  mailed  to  this  address  also. 
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BENCHMARKING  CHECKLIST:  RELIABILITY  AND  MAINTAINABILITY 

This  appendix  contains  the  benchmarking  checklist  covering  "Reliability 
and  Maintainability"  aspects  of  the  system  being  tested.  One  set  of  check¬ 
lists  must  be  completed  for  each  different  system  tested.  Check  only  one 
score  per  feature.  Indicate  your  score  by  placing  a  mark  in  the  column 
corresponding  to  the  score  you  feel  most  accurately  describes  the  feature. 
Ose  the  following  scoring  scale  and  corresponding  criteria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes ,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes ,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes ,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 
side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 
Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 
using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  pay  particular  attention  to  the  features 
marked  with  "  *  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  Implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 

Leader,  6400  Goldsboro  Road,  Bethesda,  MD  20817-5886.  Completed  checklists 
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APPENDIX  G 


'BENCHMARKING  CHECKLIST:  EASE  OF  USE  AND  MODIFICATION 


This  appendix  contains  the  benchmarking  checklist  for  assessing  the  "Ease 
of  Use  and  Modification"  of  each  system  subjected  to  benchmarking.  One  set  of 
checklists  must  be  completed  for  each  different  system  tested.  Check  only  one 
score  per  feature.  Indicate  your  score  by  placing  a  mark  in  the  column 
corresponding  to  the  score  you  feel  most  accurately  describes  the  feature. 
U3e  the  following  scoring  scale  and  corresponding  criteria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes ,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes ,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes ,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 
side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 

Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 

using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  pay  particular  attention  to  the  features 
marked  with  "  *  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 

Leader,  6400  Goldsboro  Road,  Bethesda,  MD  20817-5886.  Completed  checklists 
should  be  mailed  to  this  address  also. 
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APPENDIX  H 


BENCHMARKING  CHECKLIST:  HARDWARE  AND  SOFTWARE  FEATURES 

This  appendix  contains  the  benchmarking  checklist  for  evaluating  the 
"Hardware  and  Software"  features  offered  by  each  system  subjected  to  bench¬ 
marking.  One  set  of  checklists  must  be  completed  for  each  different  system 

tested.  Check  only  one  score  per  feature.  Indicate  your  score  by  placing  a 
mark  in  the  column  corresponding  to  the  score  you  feel  most  accurately  des¬ 
cribes  the  feature.  Use  the  following  scoring  scale  and  corresponding  crit¬ 
eria: 

1  -  No,  the  package  does  not  offer  the  feature 

2  -  Yes ,  the  package  offers  the  feature  but  with  severe  limitations 

3  -  Yes ,  the  package  offers  the  feature  with  moderate  limitations 

4  -  Yes,  the  package  offers  the  feature  with  minor  limitations 

5  -  Yes,  the  package  offers  the  feature  without  limitations 

Please  make  any  additional  comments  in  the  space  provided  on  the  check¬ 
list.  If  there  is  not  enough  room  for  your  comments,  please  use  the  reverse 

side  of  the  checklist  or  attach  a  separate  sheet  of  paper  with  the  comments. 
Make  clear  reference  to  the  function  and  feature  you  are  commenting  on  by 
using  the  same  numbering  system  or  title  used  on  the  checklist. 

As  you  perform  your  evaluations,  pay  particular  attention  to  the  features 
marked  with  "  •  "  in  the  left-hand  margin.  These  features  are  considered 
critical  to  the  implementation  of  the  system  and  deserve  special  attention 
during  benchmarking. 

Additional  copies  of  this  appendix  may  be  reproduced  locally  or  may  be 
obtained  from  the  Logistics  Management  Institute,  Attention:  DL501  Project 

Leader,  6400  Goldsboro  Road,  Bethesda,  MD  20817-5886.  Completed  checklists 
should  be  mailed  to  this  address  also. 
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APPENDIX  I 


DTIC  TR  DATA  BASE  DATA  ELEMENTS 

This  appendix  contains  a  brief  description  of  the  data  elements  extracted 
from  the  Defense  Technical  Information  Center  (DTIC)  Technical  Reports  (TR) 
data  base.  Citations  from  the  TR  data  base  were  selected  for  use  in  a  test 
data  base  available  during  benchmarking.  Citations  selected  for  the  extract 
are  unclassified  and  have  unlimited  distribution. 


-J 


nc 

Lem 

No. 

.1 

1 

OTIC  TR  DATA 

Element  Name  1 

1 

BASE  DATA 

Number  of 
Oocurr. 

ELEMENTS 

1  Description/ Comments 

1 

1 

1 

AD  Number 

1 

1 

1  AD  numbers  assigned  before 

1 

I 

1  1  Jan  75  oonsist  of  six  numeric 

1 

1 

j  characters 

1 

1 

I  AD  numbers  assigned  after 

1 

1 

1  1  Jan  75  oonsist  of  one  alpha- 

1 

1 

I  betio  and  alx  numeric  characters 

i 

2 

1 

1 

COSATI  Field/Group 

i 

1 

1 

to  8 

1 

i  An  unlimited  number  of  fields/ 

1 

1 

j  groups  can  be  assigned  to  a 

1 

1 

1 

1 

1  document 

i 

3 

1 

1 

I 

Constant 

1 

1 

i 

i 

I 

4 

1 

1 

NTIS  Document  Prices 

I 

1 

i 

|  Example: 

1 

l 

1 

i 

1  [NTIS  P]rlces[:  HC$]3.00  [MF$]0.95 

1 

5 

1 

1 

Coprorate  Source 

1 

1 

1 

to  8 

I 

|  Name  of  the  originating  agency 

1 

1 

1 

1 

!  of  the  document 

i 

6 

1 

1 

1 

Unclassified  Title 

1 

1 

1 

1 

• 

i 

i 

7 

I 

1 

Classified  Title* 

1 

1 

1 

1  This  field  will  be  blank  for 

1 

1 

1 

1 

1  all  records 

I 

8 

1 

1 

Title  Classification 

1 

1 

1 

i 

I  All  records  will  have  a  value 

1 

1 

1 

1 

1  of  "(0)"  ~  Unclassified 

i 

9 

1 

1 

Descriptive  Note/Soope 

1 

1 

1 

i 

1  A  phrase  that  denotes  the  type  o 

1 

Notes 

1 

1  of  report,  such  as  "Final", 

1 

1 

1 

1 

j  * Annual”  or  "Research  Mote" 

i 

10 

1 

1 

Personal  Author 

I 

1 

1 

to  5 

i 

1  May  contain  up  to  five  authors 

1 

1 

1  separated  by  a  delimiter.  Names 

1 

1 

1  will  appear  as  Last  name-comma- 

1 

1 

1  first  name  or  initial-space- 

1 

1 

1 

1 

1 

j  middle  name  or  initial. 

i 

11 

1 

1 

Report  Date/Origination 

i 

1 

1 

l 

1  Date  that  the  document  was 

1 

Date 

1 

1  prepared  by  the  originator, 

1 

i 

1 

I 

1  given  as  TYMMDD 

1 

12 

1 

1 

1 

Pagination 

1 

1 

1 

1 

1 

1  Left  justified,  zeroes  suppressed 

1 

13 

1 

1 

Journal  Announcement  Code 

1 

1 

1 

1  Indicates  which  TAB  or  Govern- 

1 

1 

1  ment  Announcement  announced 

1 

1 

1  this  document 

DTIC  TR  DATA  BASE  DATA  ELEMENTS  (CONTINUED) 


Element  Name 


14  I  Originating  Agency  Report 
I  Nuobers/Source  Series 
I  and  Report  Number 


15  I  Contract  Number  or 
I  Grant  Number 

I 

16  j  RDT&E  Project  Number 

I 

17  I  RDT&E  Task  Number 

I 

18/19 1  Monitor  Acronym/Series 


20  |  Report  Classification 


21  |  Supplementary  Note 


22  |  Alpha  Limitations 


23  I  Asterisked  Descriptors/ 

I  Unasterisked  Descriptors 


24  |  Descriptor  Classification 


25  !  Identifier  Set/Open-ended 
I  Terms 

I 

26  i  Identifier  Classification 


27  I  Abstract 

I 

28  i  Abstract  Classification 


29  i  Initial  Inventory 


Number  of 
Occurr. 


1  or  more 


1  or  more 


1  or  2 


1  or  2 


1  or  2 


1  to  20 


1  to  20 


1  to  20 


1  to  20 


Description/ Comments 


The  series  number  of  the  ori¬ 
ginating  agency  as  it  appears 
on  the  title  page  of  the  doc¬ 
ument 


Identifies  the  government  agency 
sponsoring  the  work 

All  test  records  have  a  value 
of  "U". 

Inofrmation  about  the  document 
for  which  no  space  was  provided. 

Indicates  the  distribution  and 
availability  of  this  document. 

The  key  subject  posting  terms 
that  have  been  selected  to  in¬ 
dicate  the  content  of  the  doc¬ 
ument  are  recorded  in  this 
field.  Key  descriptors  are 
preceded  by  an  asterisk. 

All  test  records  have  a  value 
of  »U". 

Additional  words  and  phrases  to 
describe  the  content  of  the  document 

All  test  records  have  a  value 
of  "U". 


All  test  records  have  a  value 
of  «U». 

Identifies  the  original  number 
of  copies  from  the  contributor 


v  V-V'V<\  y.v, 


DTIC  TR  DATA  BASE  DATA  ELEMENTS  (CONTINUED) 


DTIC 

Elen. 

No. 


Element  Name 


30 


Annotation 


Number  of 
Oocurr. 


Description/Comments 


:::::::::: 

1 


Contains  a  concise  paragraph 
on  the  contents  of  the  document 


31  I  Special  Indicator  I  1 

I  t 

32  I  Regrade  Category/  I 

I  Reclassification  Code*  I 


33 


Distribution  Availability 
Codes* 


34 

35 

36 

37 

38 

39 

45 

46 


I  Serial  Code/Source  Series 

I  Source  Code/Corporate 
I  Author  Code 

i  Document  Location 

I  Classification  Authority/ 
I  Classified  by* 

I  Declassification  Date* 

S  Downgrade  Date* 

I  Extended  by* 

I  Review  on* 


1 

1 

1 


47 


Reason  Code/Reason  for 
Extension* 


*  Because  all  test  data  are  unclassified  and  unrestricted , 
asterisked  test  fields  are  blank. 

Note:  In  fields  with  more  than  1  occurrence,  values 
are  separated  by  a  comma,  blank,  and  delimiter. 


APPENDIX  J 


DNA  ATLAS  FILE  DATA  ELEMENTS 

This  appendix  contains  a  brief  description  of  the  data  elements  extracted 
from  the  Defense  Nuclear  Agency  (DNA)  Automated  Technical  Library  Accession 
System  (ATLAS).  The  data  elements  extracted  from  ATLAS  are  used  to  establish 
a  test  data  base  for  benchmarking. 


DNA  ATLAS  FILE  DATA  ELEMENTS 


Elem. 

No. 

1  Element  Name 

i  Element 
j  Abbr. 

{Number  oft  Description/Comdents 

1  Occurr.  1 

1 

I  DTL  Number 

I  DTL-NO 

I 

1 

1 

1 

1 

I 

2 

1  Filler 

1 

1  FILLER 

l 

1 

1 

1 

1 

1 

1 

1 

3 

1  Status  Code 

i 

I  STATCD 

1 

1 

1 

I 

I  Indicates  status  of 

i 

1 

1  holding:  shelved,  on 

i 

i 

1 

1 

1  loan,  missing,  etc. 

1 

4 

1  Date  Coded 

1  DATECD 

1 

1 

1  Date  holding  was  input 

1 

i 

1  into  system 

5 

i  Primary  Report  Number 

1  PRI-RPT 

j 

i 

i 

1 

1 

1 

6 

1  Secondary  Report  Number 

I  SEC-RPT 
j 

i 

i 

i 

1 

1 

1 

i 

7 

|  DTIC  AD  Number 

|  D0C-AD-N0 

i 

i 

1 

I 

1  Includes  2  subfields: 

1 

i 

j  7  char.  AD  Number  and 

1 

i 

I  5  oheu*.  AD  Number 

8 

I  MIPR  Number 

I  DASA-MIPR 

i 

i 

1 

1 

1 

a 

9 

1  Origination  Date 

i 

1  ORG-DT 
j 

1 

i 

i 

1 

i 

i 

i 

10 

I  Originating  Agency 

1  ORG-AG 
| 

a 

! 

i 

1 

a 

i 

i 

11 

1  Security  Control  Number 

I  S-CTL-N 

1 

1 

1 

I 

1 

a 

i 

a 

1  Holding/Classification: 

1 

I  H- CL  ASS 

I 

1 

a 

i 

12 

I  -  CNWDI 

I  CNWDI 

1 

1 

13 

I  -  Handling 

I  HD 

1 

1 

i 

14 

I  -  Restricted  Data 

1  RDI 

1 

1 

i 

15 

1  -  Document  Classification 

1  CLASS 

1 

1 

i 

16 

1  -  Downgrade  Date 

I  DOWNG 
j 

1 

1 

1 

i 

j 

17 

1  Leaf  Count 

1  PG-LF 

j 

1 

1 

1 

1 

a 

i 

a 

18 

1  Contract  Number 

i 

1  CONTRACT-NO 

i 

i 

1 

1 

i 

i 

a 

1  Weeds: 

I 

I  WEEDS 

i 

1 

> 

i 

19 

|  -  Category 

1  CATEG 

1 

1 

i 

20 

{  -  Retention  Cycle 

1  R-CTL 

1 

1 

i 

21 

1  -  Office  Symbol 

1  0-SYM 

I 

1 

1 

1 

i 

i 

22 

i  Scope  Notes 

i 

I  SCOPE  NOTES 

1 

1 

1 

i 

1 

i 

i 

23 

1  Sub task 

1 

1  SUBTASK 

i 

1 

1 

1 

1 

a 

i 

a 

24 

t  Copy  Number/Series  Number 

i 

|  CS-1  to 

i 

I 

1  to  4 

1 

i 

1  CS-4 

1 

i 

DNA  ATLAS  FILE  DATA  ELEMENTS  (CONTINUED) 


Element  Name 


Copy/Series  Deletion  Date 


Microfiche  Copy  Number 


Microfiche  Deletion  Date 


Author 


Title 


Deletion  Message 


Element 

Abbr. 

::::::::::::: 

CS1-DT  to 
CS4-DT 

MF-1  to 
MF-4 

MF1-DT  to 
MF4-DT 

AUTHOR- 1  to 
AUTHOR-4 

TITLE- 1  to 
TITLE-4 

DELE-MESS 


Number  of 
Occurr. 

1  to  4 

1  to  4 

1  to  4 

1  to  3 

1  to  4 

1 


Description/Comments 


